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CLAIMS 

1 LA method for automatic linguistic indexing of images, said method 

2 comprising the steps of: 

3 establishing a database of statistical models using a statistical modeling 

4 process, wherein the database is accessible by the computer, and each of the statistical 

5 models represents a predetermined semantic category; 

6 associating a set of index terms with each statistical model in the database, 

7 wherein the set of index terms provides a description for the predetermined semantic 

8 category; 

9 extracting a plurality of feature vectors from an image to be indexed; 

10 statistically comparing the plurality of feature vectors extracted from the 

1 1 image to be indexed to the statistical models in the database; 

12 determining a set of statistical models from the database that are statistically 

13 similar to the plurality of feature vectors extracted from the image to be indexed; and 

14 extracting a set of statistically significant index terms from the descriptions of 

15 the set of statistical models and using the set of statistically significant indexed terms 

16 to index the image. 

1 2. The method of claim 1 further comprising the step of providing an 

2 image database of different semantic categories accessible by a computer wherein 

3 each category includes at least one image that relates to a predetermined concept. 
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3. The method of claim 1 wherein the step of extracting the plurality of 
feature vectors further includes the step of using a feature extraction algorithm. 

4. The method of claim 3 wherein the step of extracting the plurality of 
feature vectors further includes the step of partitioning the image into a plurality of 
pixel blocks. 

5. The method of claim 4 wherein the step of extracting the plurality of 
feature vectors further includes the step of extracting a six-dimensional feature vector 
from each of the pixel blocks. 

6. The method of claim 1 wherein the step of statistically comparing the 
plurality of feature vectors further includes the step using a feature comparison 
algorithm to compare the extraction feature vectors to the statistical models. 

7. The method of claim 1 wherein the step of extracting the set of 
statistically significant index terms further includes the step of using a significance 
processor to analyze the set of statistical models and the associated set of index terms 
to determine the set of statistically significant index terms. 
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1 8. The method of claim 1 wherein the step of extracting the plurality of 

2 feature vectors further includes the step of extracting multi-resolution block-based 

3 features. 

1 9. The method of claim 1 wherein the step of establishing the database of 

2 statistical models further includes the step of creating a statistical model using a two- 

3 dimensional multi-resolution hidden Markov modeling. 

1 10. The method of claim 1 wherein the step of associating the set of index 

2 terms with the statistical model further includes the step of associating index terms 

3 with the statistical models. 

1 11. The method of claim 1 wherein the step of providing the set of 

2 statistical models further includes the step of providing the set of statistical models 

3 from statistical models that exceed a predetermined statistical likelihood. 

1 12. The method of claim 1 wherein the step of extracting the set of 

2 statistically significant index terms further includes the step of selecting a set of 

3 statistically significant index terms that are included in the descriptions of the set of 

4 statistical models and are below a predetermined statistical threshold. 
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1 13. A method of automatic linguistic indexing of images using a computer 

2 system, said method comprising the steps of: 

3 generating a training image database having a plurality of training images 

4 wherein the plurality of training images represent at least one semantic category; 

5 assigning a textual description to the at least one semantic category in the 

6 training database; 

7 extracting a plurality of feature vectors from the training images using a 

8 statistical modeling process; 

9 generating statistical models using the extracted feature vectors, wherein the 

10 statistical models are associated with portions of the textual description assigned to 

11 the at least one category and including a plurality of paths, each of the plurality of 

12 paths representing a variation of the training images; 

13 storing the statistical models; and 

14 using the stored models to index and assign a textual description to an image 

15 by statistically comparing the models with a plurality of features extracted from the 

16 image to be indexed to determine the statistical similarity between the image and each 

17 model. 

1 14. The method of claim 13 wherein the step of extracting the plurality of 

2 feature vectors further includes the step of extracting multi-resolution block-based 

3 features. 
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1 15. The method of claim 13 wherein the step of generating the statistical 

2 model further includes the step of using two-dimensional multi-resolution hidden 

3 Markov modeling. 

1 16. The method of claim 13 wherein the step of associating the set of 

2 index terms with the statistical model further includes the step of manually 

3 associating index terms with the statistical model. 

1 17. The method of claim 13 wherein the step of providing the set of 

2 statistical models further includes the step of providing the set from statistical models 

3 that exceed a predetermined statistical likelihood. 

1 18. The method of claim 13 wherein the step of assigning the description 

2 to the image further includes the step of providing the description from a set of 

3 statistically significant index terms that are below a statistical threshold. 

1 19. A computer system for use in automatic linguistic indexing of images, 

2 said system comprising: 

3 a computer operative to receive an image to be indexed and assigned a textual 

4 description; 
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5 a plurality of different semantic categories disposed on said computer; 

6 a statistical modeling algorithm operative to construct a statistical model 

7 representative of each one of said plurality of different semantic categories; 

8 a database in communication with said computer for storing said plurality of 

9 different semantic categories and said statistical models, each of said different 

10 semantic categories and said statistical models having a predetermined textual 

1 1 description associated therewith; 

12 a feature extraction algorithm operative to extract a plurality of feature vectors 

13 from the image to be indexed; 

14 a feature comparison algorithm operative to statistically compare said 

15 plurality of feature vectors extracted from the image with each of said statistical 

16 models to determine statistical similarity between the image and each of said 

17 statistical models; and 

18 a text assigning algorithm operative to extract a set of statistically significant 

19 index terms from said predetermined textual descriptions associated with said 

20 statistical models wherein said set of index terms provide the textual description for 

21 the image to be indexed. 

1 20. The system of claim 19 wherein said feature comparison algorithm is 

2 further operative to extract a set of statistical models from said statistical models 
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3 wherein said extracted set of statistical models are most similar to said extracted 

4 feature vectors. 

1 21. The system of claim 20 wherein said set of statistically significant 

2 index terms is extracted from said set of statistical models. 

1 22. The system of claim 19 wherein said database is remotely located from 

2 said computer. 

1 23. The system of claim 19 wherein said computer and said database are in 

2 communication via the Internet. 

1 24. The system of claim 19 wherein said extracted features are multi- 

2 resolution block-based features extracted using said feature extraction algorithm. 

1 25. The system of claim 19 wherein said set of statistical models include 

2 statistical models corresponding to a predetermined statistical likelihood. 



26. The system of claim 19 wherein the set of statistically significant 
index terms are selected from the predetermined textual descriptions associated with 
the statistical models and are below a predetermined statistical threshold. 
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